Systems and methods for scheduling write requests for a solid state storage device

ABSTRACT

An I/O scheduler can manage and schedule write requests that are sent to a storage device in order to prevent the write requests from dominating a storage buffer of the storage device. The I/O scheduler can monitor the number of I/O requests in the storage buffer and maintain a balance between write requests and read requests in the storage buffer. The I/O scheduler monitor the latency of the storage device in processing write requests and manage the write requests sent to the storage device based on the latency. The I/O scheduler can utilize one or more write thresholds. The write thresholds can define thresholds of the number of write requests in the storage buffer and thresholds for the latency of the storage device in processing write requests. The write threshold can represent acceptable levels of write request in the storage buffer and acceptable latency for write requests.

BACKGROUND

Traditional operating system input/output schedulers are optimized formagnetic storage devices. Magnetic storage devices differ from solidstate storage (e.g. flash memory) in three main properties. First,magnetic storage devices can physically service only a single I/Orequest at a time. Second, in magnetic storage devices, read and writelatencies are approximately equal, and finally I/O latency is sensitiveto I/O ordering. In contrast, solid state storage can service multipleI/O requests truly concurrently, read requests are substantially (e.g.10×) faster than writes requests, and read latency is not sensitive toI/O ordering in the short term. Consequently, computers typically do notuse a traditional I/O scheduler with solid state storage. Instead, thecomputers send the read and write requests directly to the solid statestorage.

One drawback with this approach is that the I/O interface to the solidstate storage can be limited. Only a small number of read and writerequest can be issued to the solid state storage at a time (typically32), through the buffer of the solid state storage (e.g. Native CommandQueueing (NCQ) mechanism). As long as there is a mix of reads and writesin the buffer of the solid state storage, the solid state storage canservice read requests quickly. However, due to the higher latency (andmuch higher latency variance) of write requests, write requests candominate the buffer of the solid state storage. This starves out readrequests, and makes I/O latency of the solid state storage device appearto be much higher than it would be with fewer read requests.

BRIEF DESCRIPTION OF THE DRAWINGS

Various features of the implementations can be more fully appreciated,as the same become better understood with reference to the followingdetailed description of the implementations when considered inconnection with the accompanying figures, in which:

FIG. 1 illustrates an example of an environment in which write requestscheduling can be performed, according to various implementations.

FIG. 2A illustrates an example of a process for scheduling writerequests based on percentage of write requests in a storage buffer,according to various implementations.

FIG. 2B illustrates an example of a process for scheduling writerequests based on the latency of processing write requests by a storagedevice, according to various implementations.

FIG. 3A illustrates another example of an environment in which writerequest scheduling can be performed, according to variousimplementations.

FIG. 3B illustrates an example of a write history, according to variousimplementations.

FIGS. 4A and 4B illustrate another example of a process for schedulingwrite requests based on the percentage of write requests in a storagebuffer, according to various implementations.

FIGS. 5A-5C illustrate examples of the process of FIGS. 4A and 4B beingperformed on examples of a read queue, a write queue, and a storagebuffer, according to various implementations.

FIG. 6 illustrates another example of a process for scheduling writerequests based on the latency of processing write requests by a storagedevice, according to various implementations.

FIG. 7 illustrates another example of an environment in which writerequest scheduling can be performed, according to variousimplementations.

FIG. 8 illustrates another example of an environment in which writerequest scheduling can be performed, according to variousimplementations.

FIG. 9 illustrates another example of an environment in which writerequest scheduling can be performed, according to variousimplementations.

DETAILED DESCRIPTION

For simplicity and illustrative purposes, the principles of the presentteachings are described by referring mainly to examples of variousimplementations thereof. However, one of ordinary skill in the art wouldreadily recognize that the same principles are equally applicable to,and can be implemented in, many different types of information andsystems, and that any such variations do not depart from the true spiritand scope of the present teachings. Moreover, in the following detaileddescription, references are made to the accompanying figures, whichillustrate specific examples of various implementations. Electrical,mechanical, logical and structural changes can be made to the examplesof the various implementations without departing from the spirit andscope of the present teachings. The following detailed description is,therefore, not to be taken in a limiting sense and the scope of thepresent teachings is defined by the appended claims and theirequivalents.

Implementations of the present disclosure are directed to systems andmethods for scheduling write requests that are to be sent to a solidstate storage device. Because write requests can increase the apparentlatency of a solid state storage device, an I/O scheduler can manage thenumber and the timing of write requests that are sent to the storagedevice in order to prevent the write requests from dominating a storagebuffer of the storage device. In implementations, the I/O scheduler canmonitor the number of write requests and read requests in the storagebuffer of the storage device and maintain a balance between writerequests and read requests. For example, if the storage buffer isholding an unacceptable number of write requests relative to a size ofthe storage buffer, the I/O scheduler can hold write requests until thestorage device processes some of the write requests in the storagebuffer. The I/O scheduler can also hold write requests if the storagebuffer is holding an unacceptable number of write requests relative to atotal number of requests (read and write) in the storage buffer. As aresult, the I/O scheduler can maintain space in the storage buffer ofthe storage device for pending and future read requests, which areprocessed faster by the storage device.

In implementations, the I/O scheduler can also monitor the latency ofthe storage device in processing write requests and manage the writerequests sent to the storage device based on the latency. Based onprevious processing times, the I/O scheduler can determine the expectedlatency of the storage device. If the latency is at an unacceptablelevel, the I/O scheduler can hold write requests until the latencyreturns to an acceptable level. Accordingly, the I/O scheduler canprevent the apparent latency from reaching undesirable levels.

FIG. 1 illustrates an example of a computing environment 100 in whichwrite request scheduling can be performed, according to variousimplementations. While FIG. 1 illustrates various components containedin the computing environment 100, FIG. 1 illustrates one example of acomputing environment and additional components can be added andexisting components can be removed.

As illustrated in FIG. 1, the computing environment 100 can include acomputing device 102. The computing device 102 can be any type ofconventional computing device. For example, the computing device 102 canbe a desktop, laptop, server, etc., or mobile device, such as a smarttelephone, tablet computer, cellular telephone, personal digitalassistant, etc. The computing device 102 can include hardware resources,such as processors, memory, network hardware, storage devices, and thelike, and software resources, such as operating systems (OSes),application programs, and the like.

The computing environment 100 can also include a storage device 104 thatis coupled to the computing device 102. The storage device 104 can becoupled to the computing device 102 by an input/output (I/O) interface106. The I/O interface 106 can be any type of conventional I/Ointerface. For example, the I/O interface 106 can be any type ofconventional interface over which a computing device and a storagedevice can communicate and exchange data, such as small computer systeminterface (SCSI), serial ATA (SATA), serial-attached SCSI (ASA),Universal Serial Bus (USB), and the like. In implementations, thestorage device 104 can be configured to store data for use by thecomputing device 102 or other devices and systems. The storage device104 can be housed in the same enclosure with the computing device 102.Likewise, the storage device 104 can be housed in a separate enclosureand coupled to the computing device 102.

In implementations, the storage device 104 can include a controller 108and solid state storage 110. The controller 108 can be configured tocontrol and manage the storage and retrieval of data in the solid statestorage 110. The controller 108 can include both hardware and softwareresources to control the operations of the storage device, such ascommunicating with the computing device 102 via the I/O interface 106,writing data to the solid state storage 110, reading data from the solidstate storage 110, etc. The solid state storage 110 can includeintegrated circuit assemblies to persistently store data. For example,the solid state storage 110 can include flash memory and/or randomaccess memory (RAM) that are configured to persistently store data.

The computing device 102 can include an operating system (OS) 112. TheOS 112 can be any type of open-source or proprietary operating system.The OS 112 can include an I/O scheduler 114. While illustrated as beingpart of the OS 112, the I/O scheduler 114 (or any of the functionalityof the I/O scheduler 114) can be implemented in any other softwareprogram, implemented as a stand-alone software program, implemented as ahardware component in the computing system 102, or combinations thereof.The I/O scheduler 114 can be configured to send I/O requests to thestorage device 104. The I/O scheduler 114 can be configured to monitoran I/O cache 116 for I/O requests (e.g. read requests and writerequests) intended for the storage device 104. The I/O cache 116 can beany type of data structure or data structures capable of storing I/Orequests.

The storage device 104 can also include storage buffer 118. The storagebuffer 118 can be configured to store I/O requests that are to beprocessed by the storage device 104. The storage buffer 118 can be anytype of data structure or data structures capable of storing I/Orequests. The controller 108 can be configured to read the I/O requestsfrom the storage buffer and communicate with the solid state storage 110to perform the I/O requests. The controller 108 can be configured toutilize any type of scheduling algorithm for determining the order withwhich to process the I/O requests.

As mentioned above, due to the longer processing time of write requestscompared to read requests, the write requests can dominate the storagebuffer 118 and increase the apparent latency of the storage device 104.In implementations, the I/O scheduler 114 can be configured to manageand schedule write requests that are sent to the storage device 104 inorder to prevent the write requests from dominating the storage buffer118. In particular, the I/O scheduler 114 can be configured to monitorthe number of I/O requests in the storage buffer 118 and maintain abalance between write requests and read requests in the storage buffer118. Likewise, the I/O scheduler 114 can be configured to monitor thelatency of the storage device 104 in processing write requests andmanage the write requests sent to the storage device 104 based on thelatency.

In implementations, the I/O scheduler 114 can be configured to utilizeone or more write thresholds 120. The write threshold 120 can representacceptable levels of write requests in the storage buffer 118 andacceptable latency for write requests in order to maintain processingspeed of the storage device 104. In operation, when the I/O scheduler114 identifies a write request to be sent to the storage device, the I/Oscheduler 114 can check the current level of write requests in thestorage buffer 118 against the write thresholds 120. Likewise, the I/Oscheduler 114 can check the latency of write requests against the writethresholds 120. Based on the checks, the I/O scheduler 114 can determineto hold the write request in the I/O cache 116 or send the write requestto the storage device 104.

FIG. 2A illustrates a process 200 for scheduling write requests based onthe percentage of write requests pending in a storage device, accordingto various implementations. In the process 200, the illustrated stagesare examples and any of the illustrated stages can be removed,additional stages can be added, and the order of the illustrated stagescan be changed. Additionally, while the process 200 is described asbeing performed by the I/O scheduler 114, one or more of the stages ofthe process 200 can be performed by any hardware or software componentof the computing device 102.

In 202, the process can begin. In 204, the I/O scheduler 114 canidentify a write request to be sent to the storage device 104. Forexample, the I/O scheduler 114 can examine the I/O cache 116 anddetermine the next pending I/O request in the I/O cache 116. Based onthe examination, the I/O scheduler 114 can determine that the nextpending I/O request is a write request.

In 206, once the next pending I/O request is identified as a writerequest, the I/O scheduler 114 can determine the number of I/O requestsin the I/O cache 116 and the storage buffer 118. For example, the I/Oscheduler 114 can determine a number of write requests in the storagebuffer 118, a number of write requests in the I/O cache 116, a number ofread requests in the storage buffer 118, and a number of read requestsin the I/O cache 116. For example, the I/O scheduler 114 can maintain arecord of the number of write requests in the storage buffer 118, thenumber of write requests in the I/O cache 116, the number of readrequests in the storage buffer 118, and the number of read requests inthe I/O cache 116. While the I/O scheduler 114 can determine the numberof all I/O requests in the I/O cache 116 and the storage buffer 118, theI/O scheduler 114 can alternatively be configured to only determine thenumber of I/O requests in the I/O cache 116 and the storage buffer 118required for the process 200.

In 208, the I/O scheduler 114 can determine the percentage of writerequests that are in the storage buffer 118. The I/O scheduler 114 candetermine the percentage of write requests in the storage buffer basedon the number of write requests in the storage buffer 118, the number ofwrite requests in the I/O cache 116, the number of read requests in thestorage buffer 118, and the number of read requests in the I/O Cache116. For example, the percentage of write requests that are in thestorage buffer 118 can be the percentage of write requests relative tothe total number of I/O requests that are in the storage buffer 118. Inthis example, the percentage of write requests in the storage buffer 118can be given by the equation:

${\%\mspace{14mu}{of}\mspace{14mu}{write}\mspace{14mu}{request}} = \frac{{wr}_{sb}}{{wr}_{sb} + {rr}_{sb}}$where wr_(sb) is the number of write requests in the storage buffer 118and rr_(sb) is the number of read requests in the storage buffer 118.

Additionally, the percentage of write requests that are in the storagebuffer 118 can be the percentage of write requests relative to the totalnumber of I/O requests that can be held by the storage buffer 118. Inthis example, the percentage of write requests can be given by theequation:

${\%\mspace{14mu}{of}\mspace{14mu}{write}\mspace{14mu}{request}} = \frac{{wr}_{sb}}{\#\mspace{14mu}{of}\mspace{14mu}{slots}\mspace{14mu}{in}\mspace{14mu}{storage}\mspace{14mu}{buffer}}$where wr_(sb) is the number of write requests in the storage buffer 118and # of slots in the storage buffer is the total number of I/O requeststhat can be held by the storage buffer 118.

In 210, the I/O scheduler 114 can compare the percentage of writerequests in the storage buffer 118 to one or more thresholds. The I/Oscheduler 114 can access the write thresholds 120 and determine one ormore thresholds to compare to the percentage of write requests. The I/Oscheduler 114 can compare the percentage of write requests to the one ormore thresholds to determine if the percentage of write requests isoutside the one or more thresholds.

For example, the one or more thresholds can include a threshold for theallowed percentage of write requests among the available slots in thestorage buffer 118. Likewise, for example, the one or more thresholdscan include a threshold for the allowed percentage of write requestsamong the current I/O requests in the storage buffer 118.

In 212, if the I/O scheduler 114 determines that the percentage of writerequests is within the one or more thresholds, the I/O scheduler 114 cansend the write request to the storage device 104. The I/O scheduler 114can send the write request to the storage device 104 via the I/Ointerface 106. Once received, the storage device 104 can place the writerequest in the storage buffer 118 for processing and storage in thesolid state storage 110.

In 214, if the I/O scheduler 114 determines that the percentage of writerequests is outside one or more thresholds, the I/O scheduler 114 canhold the write request in the I/O cache 116. As such, the write requestcan be examined at a later time to determine if the storage device 104can handle additional write requests. For example, the I/O scheduler 114can leave the write request at the first position in the I/O cache 116and wait a time period before restarting the process 200 for the writerequest. Likewise, the I/O scheduler 114 can examine the I/O cache 116for read requests and send one or more read requests to the storagedevice 104. Once one or more of the read requests are sent to thestorage device 104, the I/O scheduler 114 can restart the process 200for the write request.

In 216, the process can end, repeat, or return to any point.

While process 200 is described as utilizing percentage of write requestsin the storage buffer, other metrics can be utilized in scheduling thesending of write requests to the storage device 104. For example, theI/O scheduler 114 can compare the number of write requests held by thestorage buffer 118 to one or more threshold that represent an absolutenumber of allowed write requests to be held by the storage buffer 118.

FIG. 2B illustrates an example of a process 250 for scheduling writerequests based on the latency of processing write requests by a storagedevice, according to various implementations. In process 250, theillustrated stages are examples and any of the illustrated stages can beremoved, additional stages can be added, and the order of theillustrated stages can be changed. Additionally, while the process 250is described as being performed by the I/O scheduler 114, one or more ofthe stages of the process 250 can be performed by any hardware orsoftware component of the computing device 102.

In 252, the process can begin. In 254, the I/O scheduler 114 canidentify a write request to be sent to the storage device 104. Forexample, the I/O scheduler 114 can examine the I/O cache 116 anddetermine the next pending I/O request in the I/O cache 116. Based onthe examination, the I/O scheduler 114 can determine that the nextpending I/O request is a write request.

In 256, the I/O scheduler 114 can determine a latency of the storagedevice 104 in processing write requests. For example, the I/O scheduler114 can determine the times required by the storage device 104 toprocess one or more previous write requests. For instance, the I/Oscheduler 114 can maintain a history of write requests that includestimes when write requests were sent to the storage device 104 and timeswhen the storage device 104 acknowledged that the write requests wereprocessed. The processing time for a write request can be given by thedifference in the time sent and time acknowledged. From the processingtimes, the I/O scheduler 114 can determine an expected latency of thestorage device 104 in processing write requests.

To determine the expected latency, the I/O scheduler 114 can utilize avariety of mathematical and statistical processes. For example, the I/Oscheduler 114 can determine that the expected latency is represented byone of the processing times of previous write requests. For instance,the I/O scheduler 114 can select one of the processing times of theprevious write requests, e.g. most recently processed write request, asthe expected latency.

In another example, the I/O scheduler 114 can determine the expectedlatency is represented by the mean latency of one or more previous writerequests. The I/O scheduler 114 can determine the mean latency based onthe equation:latency of write requests=1/nΣ _(i=1) ^(n)(t _(i))wherein n is the number of write requests and t_(i) is the time toprocess a write request. Likewise, for example, the I/O scheduler 114can determine the expected latency is represented by the median latencyof one or more previous write requests.

In determining the expected latency in the above examples, the I/Oscheduler 114 can utilize any number of previously processed writerequests. For example, the I/O scheduler 114 can utilize a number ofwrite requests that occurred over a time period or time window. In thisexample, as more write requests are processed, the time period or timewindow can be shifted forward in time to include recently processedwrite requests.

Likewise, in determining the expected latency in the above examples, theI/O scheduler 114 can utilize various statistical processes to selectthe processing times of the one or more previously processed writerequests. For example, the I/O scheduler 114 can utilize all theprocessing times within a time period or time window. Likewise, the I/Oscheduler 114 can utilize certain processing times that fall within arange relative to each other. The I/O scheduler 114 can remove certainprocessing times that are not representative of the other processingtimes within a time period or time window. For instance, as anon-limiting example, in a set of processing times in a time window, theI/O scheduler 114 can remove processing times that fall below the25^(th) percentile and above the 75^(th) percentile.

Likewise, for example, the I/O scheduler 114 can utilize mathematical orstatistical processes that extrapolate a future processing time based onthe past processing times of the write requests. The I/O scheduler 114can determine that the extrapolated processing time is representative ofthe expected latency. For instance, the I/O scheduler 114 can examinethe processing times for tends and extrapolate the future processingtime based on the trends. Likewise, for instance, the I/O scheduler 114can determine that the processing time is increasing or decreasing overtime and extrapolate a future processing time by using curve fittingtechniques.

While several examples are described above, the I/O scheduler 114 canutilize any mathematical and statistical process to determine anexpected latency that is representative of the storage device 104processing the write requests.

In 258, the I/O scheduler 114 can compare the latency of the storagedevice in processing write requests to one or more thresholds. The I/Oscheduler 114 can access the write thresholds 120 and determine one ormore thresholds to compare to the latency of the storage device inprocessing write requests. The I/O scheduler 114 can compare the latencyof the storage device in processing write requests to the one or morethresholds to determine if the latency is outside the one or morethresholds.

In 260, if the I/O scheduler 114 determines that the latency of thestorage device in processing write requests is within the one or morethresholds, the I/O scheduler 114 can send the write request to thestorage device 104. The I/O scheduler 114 can send the write request tothe storage device 104 via the I/O interface 106. Once received, thestorage device 104 can place the write request in the storage buffer 118for processing and storage in the solid state storage 110.

In 262, if the I/O scheduler 114 determines that the latency of thestorage device in processing the write requests is outside one or morethresholds, the I/O scheduler 114 can hold the write request to the I/Ocache 116. As such, the write request can be examine at a later time todetermine if the storage device 104 can handle additional writerequests. For example, the I/O scheduler 114 can leave the write requestat the first position in the I/O cache 116 and wait a time period beforerestarting the process 200 for the write request. Likewise, the I/Oscheduler 114 can examine the I/O cache 116 for read requests and sendone or more read requests to the storage device 104. Once one or more ofthe read requests are sent to the storage device 104, the I/O scheduler114 can restart the process 200 for the write request.

In 264, the process can end, repeat, or return to any point.

The processes 200 and 250 have been described above as separateprocesses. In implementations, both of the processes 200 and 250 can beperformed on a particular write request to determine if the particularwrite request should be sent to the storage device 104 or returned tothe I/O cache 116.

FIG. 3A illustrates another example of the computing environment 100 inwhich write request scheduling can be performed, according to variousimplementations. While FIG. 3A illustrates various components containedin the computing environment 100, FIG. 3A illustrates one example of acomputing environment and additional components can be added andexisting components can be removed.

As illustrated in FIG. 3A, the I/O cache 116 can include a read queue302 and a write queue 304. The read queue 302 can be configured to holdthe read requests intended for the storage device 104. The read queue302 can be configured in a first in first out (FIFO) structure. Thewrite queue 304 can be configured to hold the write requests intendedfor the storage device 104. The write queue 304 can be configured in afirst in first out FIFO structure.

In implementations, the I/O scheduler 114 can be configured to maintainone or more counters in order to track the number of requests in the I/Ocache 116 and the storage buffer 118. As illustrated, the I/O scheduler114 can maintain an I/O read counter 306, a storage read counter 308, anI/O write counter 310, and a storage write counter 312. The I/O readcounter 306 can be configured to track the number of read requests heldby the read queue 302. In operation, when a read request is added to theread queue 302, the I/O scheduler 114 can increment the I/O read counter306. When a read request is removed from the read queue 302 and sent tothe storage device 104, the I/O scheduler 114 can decrement the I/O readcounter 306. Likewise, the I/O write counter 310 can be configured totrack the number of write requests held by the write queue 304. Inoperation, when a write request is added to the write queue 304, the I/Oscheduler 114 can increment the I/O write counter 310. When a writerequest is removed from the write queue 304 and sent to the storagedevice 104, the I/O scheduler 114 can decrement the I/O write counter310. While the above describes using counters to track the number of I/Orequests in the I/O cache 116, the I/O scheduler 114 can utilize anytype of data structure to track the number of I/O requests in the I/Ocache 116.

The storage read counter 308 can be configured to track the number ofread requests held by the storage buffer 118. In operation, when a readrequest is sent to the storage device 104, the I/O scheduler 114 canincrement the storage read counter 308. When the I/O scheduler 114receives an acknowledgment from the storage device 104 that a readrequest has been completed, the I/O scheduler 114 can decrement thestorage read counter 308. Likewise, the storage write counter 312 can beconfigured to track the number of write requests held by the storagebuffer 118. In operation, when a write request is sent to the storagedevice 104, the I/O scheduler 114 can increment the storage writecounter 312. When the I/O scheduler 114 receives an acknowledgment fromthe storage device 104 that a write request has been completed, the I/Oscheduler 114 can decrement the storage write counter 312. While theabove describes using counters to track the number of I/O requests inthe storage buffer 118, the I/O scheduler 114 can utilize any type ofdata structure to track the number of I/O requests in the storage buffer118.

The I/O scheduler 114 can also maintain a write history 314. The I/Oscheduler 114 can utilize the write history 314 to track the writerequests sent to the storage device 104, the details of the writerequests, and the details of the processing of the write requests by thestorage device 104. The write history 314 can be implemented using anytype of data structure.

FIG. 3B illustrates an example of the write history 314, according tovarious implementations. While FIG. 3B illustrates various data fieldscontained in the write history 314, FIG. 3B illustrates one example of adata structure for the write history 314 and additional data fields canbe added and existing data fields can be removed.

As illustrated, the write history 314 can be configured as a table 350that can store details of the write requests. Each row in the table 350can correspond to a write request that has been sent to the storagedevice 104. The columns in the table can include data for each writerequest that has been sent to the storage device 104. For example, thefirst column can correspond to an identification of the write request.The second column can correspond to a time stamp of the time the writerequest was sent. The third column can correspond to a time stamp of thetime an acknowledgment of the completion of the write request wasreceived from the storage device 104.

In implementations, the I/O scheduler 114 can be configured to utilizeone or more write thresholds 316. The write thresholds 316 can definethresholds of the number of write requests in the storage buffer 118 andthresholds for the latency of the storage device 104 in processing writerequests. The write threshold 316 can represent acceptable levels ofwrite requests in the storage buffer 118 and acceptable latency forwrite requests in order to maintain processing speed of the storagedevice 104. In operation, when the I/O scheduler 114 identifies a writerequests to be sent to the storage device, the I/O scheduler 114 cancheck the current level of write requests in the storage buffer 118against the write thresholds 316. Likewise, the l/O scheduler 114 cancheck the latency of write requests against the write thresholds 316.Based on the checks, the I/O scheduler 114 can determine to hold thewrite request in the I/O cache 116 or send the write request to thestorage device 104.

In this example, the write threshold 316 can include three thresholds,Threshold 1, Threshold 2, and Threshold 3. Threshold 1 can represent theacceptable percentage of write requests, relative to total I/O requestsin the storage buffer 118, when there are read requests waiting to besent to the storage device 104. As such, the Threshold 1 can limit thenumber of write requests being sent to the storage device 104 when theread queue 302 has read requests that can be sent. Thus, the I/Oscheduler 114 can prevent write requests from filling up the storagebuffer 118 when read requests are available. Threshold 2 can representthe acceptable percentage of writes requests in the storage buffer 118relative to the size storage buffer 118. By utilizing the Threshold 2,the I/O scheduler 114 can prevent the storage buffer 118 from fillingwith write requests, for example, a large number of write requests areidentified without any read requests. Threshold 3 can represent theacceptable latency of the storage device 104 in processing writerequests. Threshold 3 can be an amount of time, for example,milliseconds, representing the acceptable latency of the storage device104 in processing write requests.

FIGS. 4A and 4B illustrate another process 400 for scheduling writerequests based on the percentage of write requests pending in a storagedevice illustrated in FIG. 3A, according to various implementations. Inthe process 400, the illustrated stages are examples and any of theillustrated stages can be removed, additional stages can be added, andthe order of the illustrated stages can be changed. Additionally, whilethe process 400 is described as being performed by the I/O scheduler114, one or more of the stages of the process 400 can be performed byany hardware or software component of the computing device 102.

In 402, the process can begin. In 404, the I/O scheduler 114 canidentify a write request to be sent to the storage device 104. Forexample, the I/O scheduler 114 can examine the I/O cache 116, inparticular, the read queue 302 and the write queue 304, and determinethe next pending I/O request in the I/O cache 116. Based on theexamination, the I/O scheduler 114 can determine that the next pendingI/O request is a write request.

In 406, once the next pending I/O request is identified as a writerequest, the I/O scheduler 114 can determine the value of the I/O readcounter 306. The value of the I/O read counter 306 indicates the numberof read requests being held by the read queue 302. In 408, the I/Oscheduler 114 can determine if the value of the I/O read counter 306 isgreater than “0.” By determining the value of the I/O read counter 306is not greater than “0,” the I/O scheduler 114 can determine that theread queue 302 does not contain a read request waiting to be sent to thestorage device. If the value of the I/O read counter 306 is not greaterthan “0,” the I/O scheduler 114 can proceed to 420.

If the value of the I/O read counter 306 is greater than “0,” the I/Oscheduler 114 can proceed to 410. By determining that the value of theI/O read counter 306 is greater than “0,” the I/O scheduler 114 candetermine that the read queue 302 contains a read request waiting to besent to the storage device. In 410-414, the I/O scheduler 114 can applyThreshold 1 which defines the acceptable percentage of write requests inthe storage buffer 118 relative to the total number of I/O requests inthe storage buffer 118.

In 410, the I/O scheduler 114 can determine the values of the storageread counter 308 and the storage write counter 312. The value of thestorage read counter 308 indicates the number of read requests beingheld by the storage buffer 118. The value of the storage write counterindicates the number of write requests being held by the storage buffer118.

In 412, the I/O scheduler 114 can determine the percentage of writerequests relative to the total number of I/O requests that are in thestorage buffer 118. The I/O scheduler 114 can determine the percentageof write requests in the storage buffer based on the number of writerequests in the storage buffer 118 and the number of read requests inthe storage buffer 118. The percentage of write requests in the storagebuffer 118 can be given by the equation:

${\%\mspace{14mu}{of}\mspace{14mu}{write}\mspace{14mu}{request}} = \frac{{WCount}_{sb}}{{WCount}_{sb} + {RCount}_{sb}}$where WCount_(sb) is the value of the storage write counter 312 andRCount_(sb) is the value of the storage read counter 308.

In 414, the I/O scheduler 114 can determine if the percentage of writerequests is greater then Threshold 1. Threshold 1 can represent themaximum desirable percentage of write requests, relative to total I/Orequests in the storage buffer 118, when there are read requests waitingto be sent to the storage device 104. As such, the Threshold 1 can limitthe number of write requests being sent to the storage device 104 whenthe read queue 302 has read requests that can be sent. Thus, the I/Oscheduler 114 can prevent write requests from filling up the storagebuffer 118 when read requests are available.

If the percentage of write requests is greater than Threshold 1, the I/Oscheduler 114 can hold the write request to the write queue 304, asshown at 416. As such, the write request can be examined at a later timeto determine if the storage device 104 can handle additional writerequests. For example, the I/O scheduler 114 can leave the write requestat the first position in the write queue 304 and wait a time periodbefore restarting the process 400 for the write request. Likewise, theI/O scheduler 114 can examine the read queue 302 for read requests andsend one or more read requests to the storage device 104. Once one ormore of the read requests are sent to the storage device 104, the I/Oscheduler 114 can restart the process 400 for the write request. In 418,the process can end, repeat, or return to any point.

If the percentage of write requests is less than or equal to Threshold1, the I/O scheduler 114 can proceed to 420 (illustrated in FIG. 4B). In420-430, the I/O scheduler 114 can apply Threshold 2 which defines theacceptable percentage of writes requests in the storage buffer 118relative to the size storage buffer 118. By utilizing the Threshold 2,the I/O scheduler 114 can prevent the storage buffer 118 from fillingwith write requests, for example, a large number of write requests areidentified without any read requests.

In 420, the I/O scheduler 114 can determine the percentage of writerequests in the storage buffer 118 relative to the size of the storagebuffer 118. Additionally, the size of the storage buffer 118 can be thetotal number of I/O requests that can be held by the storage buffer 118.The percentage of write requests can be given by the equation:

${\%\mspace{14mu}{of}\mspace{14mu}{write}\mspace{14mu}{request}} = \frac{{WCount}_{sb}}{\#\mspace{14mu}{of}\mspace{14mu}{slots}\mspace{14mu}{in}\mspace{14mu}{storage}\mspace{14mu}{buffer}}$where WCount_(sb) is the number of write requests in the storage buffer118 and # of slots in the storage buffer is the total number of I/Orequests that can be held by the storage buffer 118.

In 422, the I/O scheduler 114 can compare the percentage of writerequests in the storage buffer 118 to Threshold 2. In 424, if thepercentage of write requests is greater than Threshold 2, the I/Oscheduler 114 can hold the write request to the write queue 304. Assuch, the write request can be examined at a later time to determine ifthe storage device 104 can handle additional write requests. Forexample, the I/O scheduler 114 can leave the write request at the firstposition in the write queue 304 and wait a time period before restartingthe process 400 for the write request. Likewise, the I/O scheduler 114can examine the read queue 302 for read requests and send one or moreread requests to the storage device 104.

In 426, if the I/O scheduler 114 determines that the percentage of writerequests is less than or equal to Threshold 2, the I/O scheduler 114 cansend the write request to the storage device 104. The I/O scheduler 114can send the write request to the storage device 104 via the I/Ointerface 106. Once received, the storage device 104 can place the writerequest in the storage buffer 118 for processing and storage in thesolid state storage 110.

In 428, the I/O scheduler 114 can increment the storage write counter312 and can decrement the I/O write counter 310. As such, the I/Oscheduler 114 can maintain an accurate count of the I/O requests in theI/O cache 116 and the storage buffer 118.

In 430, the process can end, repeat, or return to any point.

FIGS. 5A-5C illustrate an example of the process 400 being performed onan example of the storage buffer 118, the read queue 302, and the writequeue 304. While FIGS. 5A-5C illustrate an example of the storage buffer118, the read queue 302, and the write queue 304, the storage buffer118, the read queue 302, and the write queue 304 can be configured inany structure and size.

As illustrated in 5A, the storage buffer 118 can be holding 3 writerequests, the read queue 302 can be holding 1 read request, and thewrite queue 304 can be holding 2 write requests, which are reflected inthe I/O read counter 306, the storage read counter 308, the I/O writecounter 310, and the storage write 312. In this example, the Threshold 1can be 30% and the Threshold 2 can be 50%.

The I/O scheduler 114 can begin the process 400 and identify that thewrite queue 304 is holding a write request to be sent to the storagedevice 104. Then, the I/O scheduler 114 can read the I/O read counter306 and determine that the read queue 302 is holding a read request.Then, the I/O scheduler 114 can determine the percentage of writerequests relative to the total number of I/O requests that are in thestorage buffer 118 based on the values stored in the storage readcounter 308 and the storage write 312. In this example, the I/Oscheduler 114 can determine that the percentage is 100% (i.e., onlywrite requests in the storage buffer 118.) The I/O scheduler 114 canthen compare this percentage to the Threshold 1 and determine that thewrite request should be held in the write queue 304. Because the writerequest is being held, the I/O scheduler can send the read request tothe storage device 104. The I/O scheduler 114 can also update the I/Oread counter 306 and the storage read counter 308.

As illustrated in FIG. 5B, the I/O scheduler can restart the process 400to determine if the write request can be sent to the storage device 104.In this example, the I/O scheduler can determine that the read queue 302is not holding any read requests. Accordingly, the I/O scheduler 114 candetermine the percentage of write requests relative to the size of thestorage buffer 118 based on the values stored in the storage write 312.In this example, the I/O scheduler 114 can determine that the percentageis 60%. The I/O scheduler 114 can then compare this percentage to theThreshold 2 and determine that the write request should be held in thewrite queue 304.

As illustrated in FIG. 5C, the I/O scheduler can restart the process 400to determine if the write request can be sent to the storage device 104.Since the last execution of process 400, the storage device completedone of the write requests in the storage buffer 118. In this example,the I/O scheduler can determine that the read queue 302 is not holdingany read requests. Accordingly, the I/O scheduler 114 can determine thepercentage of write requests relative to the size of the storage buffer118 based on the values stored in the storage write 312. In thisexample, the I/O scheduler 114 can determine that the percentage is 40%.The I/O scheduler 114 can then compare this percentage to the Threshold2 and determine that the write request can be sent to the storage device104. The I/O scheduler 114 can also update the I/O write counter 310 andthe storage write counter 312.

FIG. 6 illustrates an example of a process 600 for scheduling writerequests based on the latency of processing write requests by a storagedevice illustrated in FIG. 3A, according to various implementations. Inprocess 600, the illustrated stages are examples and any of theillustrated stages can be removed, additional stages can be added, andthe order of the illustrated stages can be changed. Additionally, whilethe process 600 is described as being performed by the I/O scheduler114, one or more of the stages of the process 600 can be performed byany hardware or software component of the computing device 100.

In 602, the process can begin. In 604, the I/O scheduler 114 canidentify a write request to be sent to the storage device 104. Forexample, the I/O scheduler 114 can examine the I/O cache 116, inparticular, the read queue 302 and the write queue 304, and determinethe next pending I/O request in the I/O cache 116. Based on theexamination, the I/O scheduler 114 can determine that the next pendingI/O request is a write request.

In 606, once the next pending request is identified as a write request,the I/O scheduler 114 can determine the latency of the storage device104 in processing write requests from the write history 314. Forexample, the I/O scheduler 114 can determine the times required by thestorage device 104 to process one or more previous write requests. Forinstance, the I/O scheduler 114 can extract, from the write history 314,the times when write requests were sent to the storage device 104 andthe times when the storage device 104 acknowledged that the writerequests were processed. The processing time for a write request can begiven by the difference in the time sent and time acknowledged. From theprocessing times, the I/O scheduler 114 can determine an expectedlatency of the storage device 104 in processing write requests.

To determine the expected latency, the I/O scheduler 114 can utilize avariety of mathematical and statistical processes. For example, the I/Oscheduler 114 can determine that the expected latency is represented byone of the processing times of previous write requests. For instance,the I/O scheduler 114 can select one of the processing times of theprevious write requests, e.g. most recently processed write request, asthe expected latency.

In another example, the I/O scheduler 114 can determine the expectedlatency is represented by the mean latency of one or more previous writerequests. The I/O scheduler 114 can determine the mean latency based onthe equation:latency of write requests=1/nΣ _(i=1) ^(n)(t _(i))wherein n is the number of write requests and t_(i) is the time toprocess a write request. Likewise, for example, the I/O scheduler 114can determine the expected latency is represented by the median latencyof one or more previous write requests.

In determining the expected latency in the above examples, the I/Oscheduler 114 can utilize any number of previously processed writerequests. For example, the I/O scheduler 114 can utilize a number ofwrite requests that occurred over a time period or time window. In thisexample, as more write requests are processed, the time period or timewindow can be shifted forward in time to include recently processedwrite requests.

Likewise, in determining the expected latency in the above examples, theI/O scheduler 114 can utilize various statistical processes to selectthe processing times of the one or more previously processed writerequests. For example, the I/O scheduler 114 can utilize all theprocessing times within a time period or time window. Likewise, the I/Oscheduler 114 can utilize certain processing times that fall within arange relative to each other. The I/O scheduler 114 can remove certainprocessing times that are not representative of the other processingtimes within a time period or time window. For instance, as anon-limiting example, in a set of processing times in a time window, theI/O scheduler 114 can remove processing times that fall below the25^(th) percentile and above the 75^(th) percentile.

Likewise, for example, the I/O scheduler 114 can utilize mathematical orstatistical processes that extrapolate a future processing time based onthe past processing times of the write requests. The I/O scheduler 114can determine that the extrapolated processing time is representative ofthe expected latency. For instance, the I/O scheduler 114 can examinethe processing times for tends and extrapolate the future processingtime based on the trends. Likewise, for instance, the I/O scheduler 114can determine that the processing time is increasing or decreasing overtime and extrapolate a future processing time by using curve fittingtechniques.

While several examples are described above, the I/O scheduler 114 canutilize any mathematical and statistical process to determine anexpected latency that is representative of the storage device 104processing the write requests.

In 608, the I/O scheduler 114 can compare the latency of the storagedevice in processing write requests to Threshold 3. The I/O scheduler114 can access the write thresholds 316 and determine the value forThreshold 3.

In 608, if the I/O scheduler 114 determines that the latency of thestorage device in processing write requests is less than or equal toThreshold 3, the I/O scheduler 114 can send the write request to thestorage device 104, as shown at 610. The I/O scheduler 114 can send thewrite request to the storage device 104 via the I/O interface 106. Oncereceived, the storage device 104 can place the write request in thestorage buffer 118 for processing and storage in the solid state storage110.

In 608, if the I/O scheduler 114 determines that the latency of thestorage device in processing the write requests is greater thanThreshold 3, the I/O scheduler 114 can hold the write request in thewrite queue 304, as shown at 612. As such, the write request can beexamined at a later time to determine if the storage device 104 canhandle additional write requests. For example, the I/O scheduler 114 canleave the write request at the first position in the write queue 304 andwait a time period before restarting the process 600 for the writerequest. Likewise, the I/O scheduler 114 can examine the read queue 302for read requests and send one or more read requests to the storagedevice 104.

In 614, the process can end, repeat, or return to any point.

The processes 400 and 600 have been described above as separateprocesses. In implementations, both the processes 400 and 600 can beperformed on a particular write request to determine if the particularwrite request should be sent to the storage device 104 or returned tothe I/O cache 116.

FIG. 7 illustrates another example of the computing environment 100 inwhich write request scheduling can be performed, according to variousimplementations. While FIG. 7 illustrates various components containedin the computing environment 100, FIG. 7 illustrates one example of acomputing environment and additional components can be added andexisting components can be removed.

As illustrated in FIG. 7, the I/O cache 116 can include a read queue 702and a write queue 704. The read queue 702 can be configured to hold theread requests intended for the storage device 104. The read queue 702can be configured in FIFO structure. The write queue 704 can beconfigured to hold the write requests intended for the storage device104. The write queue 704 can be configured in FIFO structure.

The I/O cache 116 can also include a priority queue 706. The priorityqueue 706 can be configured hold I/O requests that have a higherpriority than I/O requests in the read queue 702 and write queue 704.For example, one or more software programs implemented on the computingdevice 102 can have a higher priority then other software programsimplemented on the computing device 102. Accordingly, the I/O requestsof the higher priority software programs can take precedence over otherI/O requests. The priority queue 706 can be configured in a FIFOstructure.

In implementations, the I/O scheduler 114 can be configured to maintainone or more counters in order to track the number of requests in the I/Ocache 116 and the storage buffer 118. As illustrated, the I/O scheduler114 can maintain an I/O read counter 708, a storage read counter 710, anI/O write counter 712, and a storage write counter 714. The I/O readcounter 708 can be configured to track the number of read requests heldby the read queue 702. In operation, when a read request is added to theread queue 702, the I/O scheduler 114 can increment the I/O read counter708. When a read request is removed from the read queue 702 and sent tothe storage device 104, the I/O scheduler 114 can decrement the I/O readcounter 708. Likewise, the I/O write counter 712 can be configured totrack the number of write requests held by the write queue 704. Inoperation, when a write request is added to the write queue 704, the I/Oscheduler 114 can increment the I/O write counter 712. When a writerequest is removed from the write queue 704 and sent to the storagedevice 104, the I/O scheduler 114 can decrement the I/O write counter712.

The storage read counter 710 can be configured to track the number ofread requests held by the storage buffer 118. In operation, when a readrequest is sent to the storage device 104, the I/O scheduler 114 canincrement the storage read counter 710. When the I/O scheduler 114receives an acknowledgment from the storage device 104 that a readrequest has been completed, the I/O scheduler 114 can decrement thestorage read counter 710. Likewise, the storage write counter 714 can beconfigured to track the number of write requests held by the storagebuffer 118. In operation, when a write request is sent to the storagedevice 104, the I/O scheduler 114 can increment the storage writecounter 714. When the I/O scheduler 114 receives an acknowledgment fromthe storage device 104 that a write request has been completed, the I/Oscheduler 114 can decrement the storage write counter 714.

The I/O scheduler 114 can also maintain a write history 716. The I/Oscheduler 114 can utilize the write history 716 to track the writerequests send to the storage device 104, the details of the writerequests, and the details of the processing of the write requests by thestorage device 104. The write history 716 can be implemented using anytype of data structure.

In implementations, the I/O scheduler 114 can be configured to processthe I/O requests in the priority queue 706 without applying theprocesses described above. That is, when an I/O request arrives in thepriority queue 706, the I/O scheduler 114 can send the I/O request tothe storage device without applying the write thresholds 120. In thisexample, the I/O read counter 708 and the I/O write counter 712 canreflect the I/O requests in the read queue 702 and the write queue 704.Alternatively, in some implementations, the I/O scheduler 114 can beconfigured to process the I/O requests in the priority queue 706 byapplying the processes described above. In this example, the I/O readcounter 708 and the I/O write counter 712 can reflect the I/O requestsin the read queue 702, the write queue 704, and the priority queue 706.

In implementations, the I/O scheduler 114 can be configured to utilizethe priority queue 706 to prioritize write requests held in the writequeue 704 and read requests held in the read queue 702. For example, theI/O scheduler 114 can maintain a deadline for sending read requests andwrite requests to the storage device 104. The deadline can represent alimit on an amount of time a read request or a write request should beheld in the I/O cache 116 before sending to the storage device 104. Oncea particular read request or a particular write request has reached thedeadline, the I/O scheduler 114 can move the particular read request orthe particular write request to the priority queue 706. Once moved tothe priority queue, the I/O scheduler 114 can process the particularread request or the particular write request at a higher priority thanother requests in the read queue 702 or the write queue 704.

FIG. 8 illustrates another example of the computing environment 100 inwhich write request scheduling can be performed, according to variousimplementations. While FIG. 8 illustrates various components containedin the computing environment 100, FIG. 8 illustrates one example of acomputing environment and additional components can be added andexisting components can be removed.

As illustrated in FIG. 8, two storage device, storage device “A” 802 andstorage device “B” 804 can be coupled to the computing device 102 viathe I/O interface 106. As discussed above, the storage device “A” 802and storage device “B” 804 can each include a controller and solid statestorage. The controller for each storage device can be configured tocontrol and manage the storage of data to the solid state storage ineach. The controller for each storage device can include both hardwareand software resources to control the operation of the storage devicesuch as communicating with the computing device 102 via the I/Ointerface 106, writing data to the solid state storage, reading datafrom the solid state storage, etc. The solid state storage for eachstorage device can include integrated circuit assemblies to persistentlystore data. For example, the solid state storage for each device caninclude flash memory and/or RAM that are configured to persistentlystore data.

In implementations, the I/O scheduler 114 can maintain an I/O cache foreach of the storage device “A” 802 and the storage device “B” 804. AnI/O cache 806 can be configured to hold I/O requests intended for thestorage device “A” 802. An I/O cache 808 can be configured to hold I/Orequests intended for the storage device “B” 804. The I/O cache 806 andthe I/O cache 808 can be any type of data structure or data structurescapable of storing I/O requests. For example, the I/O cache 806 and theI/O cache 808 can be configured as described above in the variousexamples of the computing environment 100. While the I/O scheduler 114can maintain separate I/O caches for the storage device “A” 802 and thestorage device “B” 804, the I/O scheduler 114 can maintain a single I/Ocache for the storage device “A” 802 and the storage device “B” 804.

In implementations, the I/O scheduler 114 can maintain write thresholdsfor each of the storage device “A” 802 and the storage device “B” 804.The I/O scheduler 114 can maintain a write thresholds device “A” 810.When performing any of the processes described above for the storagedevice “A” 802, the I/O scheduler 114 can utilize the write thresholdsdevice “A” 810. The I/O scheduler 114 can maintain a write thresholdsdevice “B” 812. When performing any of the processes described above forthe storage device “B” 804, the I/O scheduler 114 can utilize the writethresholds device “B” 812. While the I/O scheduler 114 can maintainseparate write thresholds for the storage device “A” 802 and the storagedevice “B” 804, the I/O scheduler 114 can maintain a single writethresholds for the storage device “A” 802 and the storage device “B”804.

In the implementations described above, the write thresholds 120 can bepreconfigured in the I/O scheduler 114. For example, the writethresholds 120 can be determined and preconfigured in the I/O scheduler114 based on known write levels and latency for different storagedevices. Likewise, the implementations described above, the writethresholds 120 can be altered or set by a user of the computing device102.

In the implementations described above, the write thresholds 120 can bedynamically altered during and/or after the operations of the processdescribed above. For example, the I/O scheduler 114 can set an initiallyvalue for a write threshold. Over time, the I/O scheduler 114 canincrease the value of the write threshold. As write requests areprocessed by the storage device 104 under the increased threshold, theI/O scheduler 114 can monitor the latency of the storage device inprocessing the write requests. If the latency remains at an acceptablelevel, the I/O scheduler 114 can continue to increase the value of thewrite threshold. If the latency reaches an unacceptable level, the I/Oscheduler 114 can decrease the value of the write threshold to aprevious level.

FIG. 9 illustrates an example of a hardware configuration for acomputing device 102 implementing the I/O scheduler 114 that can be usedto perform one or more of the processes described above. While FIG. 9illustrates various components contained in the computing device 102,FIG. 9 illustrates one example of a computing device and additionalcomponents can be added and existing components can be removed.

As illustrated in FIG. 9, the computing device 102 can include one ormore processors 902 of varying core configurations and clockfrequencies. The computing device 102 can also include one or morememory devices 904 that serve as a main memory during the operation ofthe computing device 102. For example, during operation, a copy of theOS 112 and the I/O scheduler 114 can be stored in the one or more memorydevices 904. Additionally, during operation, the I/O cache 116 can beimplemented in the one or more memory devices 904. The computing device102 can also include one or more peripheral interfaces 906, such askeyboards, mice, touchpads, computer screens, touchscreens, etc., forenabling human interaction with and manipulation of the computing device102.

The computing device 102 can also include one or more network interfaces908 for communicating via one or more networks, such as Ethernetadapters, wireless transceivers, or serial network components, forcommunicating over wired or wireless media using protocols. Thecomputing device 102 can also include one or more storage device 910 ofvarying physical dimensions and storage capacities, such as flashdrives, hard drives, random access memory, etc., for storing data, suchas images, files, and program instructions for execution by the one ormore processors 902. For example, the one or more storage devices 910can include the storage device 102.

Additionally, the computing device 102 can include one or more softwareprograms 912, such as the OS 112 and the I/O scheduler 114. The one ormore software programs 912 can include instructions that cause the oneor more processors 902 to perform the processes described herein. Copiesof the one or more software programs 912 can be stored in the one ormore memory devices 904 and/or on in the one or more storage devices910. Likewise, the data, for example, the write history 314, utilized byone or more software programs 912 can be stored in the one or morememory devices 904 and/or on in the one or more storage devices 910.

In implementations, the computing device 102 can communicate with one ormore remote computing devices 914 via a network 916. The one or moreremote computing devices 914 can be any types of conventional computingdevices. For example, the one or more remote computing devices 914 canbe desktops, laptops, servers, etc., or mobile devices, such as smarttelephones, tablet computers, cellular telephones, personal digitalassistants, etc. The network 916 can be any type of network, such as alocal area network, a wide-area network, a virtual private network, theInternet, an intranet, an extranet, a public switched telephone network,an infrared network, a wireless network, and any combination thereof.The network 916 can support communications using any of a variety ofcommercially-available protocols, such as TCP/IP, OSI, FTP, UPnP, NFS,CIFS, and AppleTalk. The network 916 can be, for example, a local areanetwork, a wide-area network, a virtual private network, the Internet,an intranet, an extranet, a public switched telephone network, aninfrared network, a wireless network, and any combination thereof.

In implementations, the computing device 102 can exchange data with theone or more remote computing devices 914 over the network 916. Forexample, the computing device 102 can receive I/O requests for thestorage device 104 from the one or more remote computing devices 914.Accordingly, the computing device 102 can utilize the processesdescribed above when processing the I/O requests from the one or moreremote computing devices 914. Likewise, the computing device 102 canreceive one or more of the write thresholds 120 from the one or moreremote computing devices 914.

The computing device 102 and the storage device 104 also can beimplemented as part of at least one service or Web service, such as maybe part of a service-oriented architecture. For example, the computingdevice 102 can exchange data with the one or more remote computingsystems 914 during operation of the at least one service or Web service.Services such as Web services can communicate using any appropriate typeof messaging, such as by using messages in extensible markup language(XML) format and exchanged using an appropriate protocol such as SOAP(derived from the “Simple Object Access Protocol”). Processes providedor executed by such services can be written in any appropriate language,such as the Web Services Description Language (WSDL). Using a languagesuch as WSDL allows for functionality such as the automated generationof client-side code in various SOAP frameworks.

In implementations the computing device 102 and the storage device 104can be utilized as part of a Web server architecture. In the Web serverarchitecture, the computing device 102 can run any of a variety ofserver or mid-tier applications, including HTTP servers, FTP servers,CGI servers, data servers, Java servers, and business applicationservers. The computing device 102 also can be capable of executingprograms or scripts in response requests from the one or more remotecomputing devices 914, such as by executing one or more Web applicationsthat may be implemented as one or more scripts or programs written inany programming language, such as Java®, C, C# or C++, or any scriptinglanguage, such as Perl, Python, or TCL, as well as combinations thereof.The computing device 102 can also include database servers, includingwithout limitation those commercially available from Oracle®,Microsoft®, Sybase®, and IBM®.

The computing environment 100 can include a variety of data stores andother memory and storage media as discussed above. These can reside in avariety of locations, such as on a storage medium local to (and/orresident in) one or more of the computers or remote from any or all ofthe computers across the network. In some implementations, informationcan reside in a storage-area network (“SAN”) familiar to those skilledin the art. Similarly, any necessary files for performing the functionsattributed to the computers, servers, or other network devices may bestored locally and/or remotely, as appropriate.

In implementations, the components of the computing device 102 asdescribed above need not be enclosed within a single enclosure or evenlocated in close proximity to one another. Those skilled in the art willappreciate that the above-described componentry are examples only, asthe computing device 102 can include any type of hardware componentry,including any necessary accompanying firmware or software, forperforming the disclosed implementations. The computing device 102 canalso be implemented in part or in whole by electronic circuit componentsor processors, such as application-specific integrated circuits (ASICs)or field-programmable gate arrays (FPGAs).

Certain implementations described above can be performed as a computerapplications or programs. The computer program can exist in a variety offorms both active and inactive. For example, the computer program canexist as one or more software programs, software modules, or both thatcan be comprised of program instructions in source code, object code,executable code or other formats; firmware program(s); or hardwaredescription language (HDL) files. Any of the above can be embodied on acomputer readable medium, which include non-transitory computer readablestorage devices and media, and signals, in compressed or uncompressedform. Examples of non-transitory computer readable storage devices andmedia include conventional computer system RAM (random access memory),ROM (read-only memory), EPROM (erasable, programmable ROM), EEPROM(electrically erasable, programmable ROM), and solid state, magnetic oroptical disks or tapes. Examples of computer readable signals, whethermodulated using a carrier or not, are signals that a computer systemhosting or running the present teachings can be configured to access,including signals downloaded through the Internet or other networks.Concrete examples of the foregoing include distribution of executablesoftware program(s) of the computer program on a CD-ROM or via Internetdownload. In a sense, the Internet itself, as an abstract entity, is acomputer readable medium. The same is true of computer networks ingeneral.

While the teachings have been described with reference to examples ofthe implementations thereof, those skilled in the art will be able tomake various modifications to the described implementations withoutdeparting from the true spirit and scope. The terms and descriptionsused herein are set forth by way of illustration only and are not meantas limitations. In particular, although the method has been described byexamples, the steps of the method may be performed in a different orderthan illustrated or simultaneously. Furthermore, to the extent that theterms “including”, “includes”, “having”, “has”, “with”, or variantsthereof are used in either the detailed description and the claims, suchterms are intended to be inclusive in a manner similar to the term“comprising.” As used herein, the terms “one or more of” and “at leastone of” with respect to a listing of items such as, for example, A andB, means A alone, B alone, or A and B. Further, unless specifiedotherwise, the term “set” should be interpreted as “one or more.” Also,the term “couple” or “couples” is intended to mean either an indirect ordirect connection. Thus, if a first device couples to a second device,that connection may be through a direct connection, or through anindirect connection via other devices, components, and connections.

What is claimed is:
 1. A non-transitory computer readable storage mediumhaving stored thereon instructions that when executed by one or morecomputing systems cause the one or more computing systems to: identify awrite request to be sent to a storage device; determine, from a firstcounter, that a read request is being held in a first cache waiting tobe sent to the storage device; determine, from a second counter, anumber of read requests that are being held in a buffer of the storagedevice; determine, from a third counter, a number of write requests thatare being held in the buffer of the storage device; determine a firstpercentage of write requests among a total of the number of readrequests and the number of the write requests that are being held in thebuffer of the storage device; compare the first percentage to a firstthreshold; in response to the first percentage being less than or equalto the first threshold, send the write request to the storage device;and in response to the first percentage being greater than the firstthreshold, hold the write request in a second cache for sending to thestorage device at a later time.
 2. The non-transitory computer readablestorage medium of claim 1, wherein the method, in response to the firstpercentage being less than or equal to the first threshold, furthercomprises: determine, prior to sending the write request to the storage,a total number of input/output (I/O) requests that can be held by thebuffer of the storage device; determine, prior to sending the writerequest to the storage, a second percentage of write requests among thetotal number of I/O requests that can be held by the buffer of thestorage device; compare the second percentage to a second threshold; inresponse to the second percentage being less than or equal to the secondthreshold, send the write request to the storage device; and in responseto the second percentage being greater than the second threshold, holdthe write request the second cache for sending to the storage device ata later time.
 3. The non-transitory computer readable storage medium ofclaim 2, wherein at least one of the first threshold and the secondthreshold is dynamically adjusted.
 4. A computer-implemented method ofmanaging storage, the method comprising: identifying a write request tobe sent to a storage device; determining a number of write requests thatare being held by the storage device; determining, based at least inpart on the number of write requests, a percentage of write requestsamong a total number of requests that can be held by the storage device;comparing, using a processor, the percentage of write requests to athreshold, in response to the percentage being less than or equal to thethreshold, sending the write request to the storage device; and inresponse to the percentage being greater than the threshold, holding thewrite request for sending to the storage device at a later time.
 5. Thecomputer-implemented method of claim 4, the method further comprising:determining, prior to comparing the percentage to the threshold, that atleast one read request is waiting to be sent to the storage device;determining a number of read request that are being held by the storagedevice; determining an additional percentage of write requests among atotal of the number of write requests and the number of read requeststhat are being held by the storage device; comparing the additionalpercentage to an additional threshold; and determining that that theadditional percentage is less than or equal to the additional threshold.6. The method of claim 5, the method further comprising: dynamicallyadjusting at least one of the threshold and the additional thresholdbased at least in part on an amount of time required by the storagedevice to complete write requests.
 7. The method of claim 6, whereindynamically adjusting at least one of the threshold and the additionalthreshold, comprises: determining that the amount of time required bythe storage device to complete write requests is less than apredetermined amount of time; and increasing at least one of thethreshold and the additional threshold in response to the amount of timebeing less than the predetermined amount of time.
 8. The method of claim5, the method further comprising: receiving, via a network, at least oneof the threshold and the additional threshold.
 9. The method of claim 4,the method further comprising: receiving, via a network, the writerequest.
 10. The method of claim 4, the method further comprising:identifying, prior to determining the number of write requests, apriority I/O request that is waiting to be sent to the storage device;and sending, prior to determining the number of write requests, thepriority I/O request to the storage device.
 11. The method of claim 4,the method further comprising: determining, in response to holding thewrite request, that the write request has been held beyond a deadline tosend the write request to the storage device; and determining that thewrite request is a priority I/O request.
 12. A system for managingstorage, the system comprising: at least one solid state storage device;and at least one processor coupled to the at least one solid statestorage device and configured to perform a method comprising:identifying a write request to be sent to at least one solid statestorage device; determining an expected amount of time required by theat least one solid state storage device to complete write requests;comparing the expected amount of time to a threshold; in response to theexpected amount of time being less than or equal to the threshold,sending the write request to the at least one solid state storagedevice; and in response to the expected amount of time being equal to orgreater than the threshold, holding the write request for sending to theat least one solid state storage device at a later time.
 13. The systemof claim 12, wherein determining the expected amount of time comprises:determining, for one or more previously sent write requests, one or moreactual amounts of time required to complete the one or more previouslysent write requests; and determining the expected amount of time basedat least in part on the one or more actual amounts of time.
 14. Thesystem of claim 13, wherein determining the expected amount of timecomprises: determining a mean value of the one or more actual amounts oftime.
 15. The system of claim 13, wherein at least one of the one ormore actual amounts of time is weighted higher in the determination ofthe expected amount of time.
 16. The system of claim 13, whereindetermining the amount of time comprises: removing, from thedetermination of the expected amount of time, at least one of the one ormore actual amounts of time that differs from others of the one or moreactual amounts of time.
 17. The system of claim 13, wherein determiningthe expected amount of time comprises: selecting the one or morepreviously sent write requests that occur within a period of time. 18.The system of claim 13, the method further comprising: storing the oneor more actual amounts of time in a history of previously sent writerequests.
 19. The system of claim 12, the system further comprising: anetwork interface coupled to the at least one processor, wherein the atleast one solid state storage device provides network-based storage forone or more remote computing devices.
 20. The system of claim 19,wherein the at least one processor is further configured to perform themethod further comprising: receiving, via the network interface, thewrite request.
 21. The system of claim 12, wherein the at least oneprocessor is further configured to perform the method furthercomprising: dynamically adjusting the threshold.